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Overview 


Aw the country, high school teachers are being called upon to improve student writing, but they often lack the 
tools and requisite know-how to make a difference. An ambitious new program called Drive to Write is attempting 
to change that. This report describes an evaluation of the program's implementation in 11 public high schools in New York 
City during the 2017-2018 school year. 


Drive to Write involves expert coaches to help teachers use Google Suite tools for managing classroom assignments, 
providing actionable feedback to students, and using data to assess a student’s progress for the sake of differentiating, 
or customizing, instruction. The program was designed by New Visions for Public Schools, a support network in 
New York City that offers services such as professional development and data infrastructure to public schools. The 
intervention targets ninth grade Global History teachers who are not trained writing instructors. Their students are 
scheduled to take the Global History Regents Exam at the end of tenth grade — a prerequisite for graduation. 


As of 2016, only 68 percent of students passed this exam. In June 2019, the test is likely to become even harder because 
of a new, rigorous essay requirement that accounts for almost one-third of the total score. The Drive to Write pro- 
gram focuses on this essay to help teachers advance student writing. 


Key Findings 


e The program rolled out as intended throughout the 2017-2018 school year. Coaches tailored their feedback for 
teachers and helped them focus on writing instruction by using technology to support workflow and data to guide 
their approach to individual students. Overall, teachers expressed a high rate of satisfaction with, and adoption of, 
the tools and support provided by Drive to Write. 


¢ Teachers customized their use of technology tools and writing instruction to suit the needs of their students and 
the constraints of their classroom. Nevertheless, practices related to writing and technology use among the 15 
program teachers in the 11 Drive to Write schools were similar to those of the 17 teachers in 12 comparison schools. 
Teachers in program schools, however, exhibited greater understanding of, and proficiency with, higher-level writ- 
ing instruction. 


e It is unclear whether the program had a positive effect on student writing after one academic year of implemen- 
tation. The analytic sample included 1,008 program students and 936 comparison students. Several factors could 
have dampened early effects, such as comparable writing improvement among all students during ninth-grade, 
similar technology practices between program and comparison students, or a sample of schools too small to detect 
modest effects. It could also be that the assessment score outcome may reflect student skill at timed test taking 
(which all schools address), rather than the intervention’s core focus on intensive writing composition (on which 
program schools spent dedicated time). 


This evaluation contributes to the growing literature that highlights the support teachers require to integrate new 
technology and data tools into their instructional routines, the role of individualized coaching for teachers, and the 
sustainability of data-driven teacher feedback to students. An understanding of these elements can lead to better 
implementation of writing programs in high schools across the country and, potentially, improved student writing. 
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Introduction 


n Ms. B’s classroom at a small high school in New York City, class begins when students turn on their computers. 

Before dispersing into smaller groups, the 30 students in this ninth-grade Global History class grab laptops from a 
cart and log in to their individual Google Classroom accounts. They open a draft of the essay they have been working 
on for the past four days and begin addressing Ms. B’s comments. Some are rewriting sentences, others are adding 
primary source documents. Ms. B weaves between desks, answering questions. One student needs help displaying 
teacher comments in his document. Another is struggling with essay structure. Ms. B reminds some students of 
earlier lessons that used building blocks to model an hourglass essay construction. The lesson plan for the week, and 
the year, is to help students improve their writing, specifically the type of essay writing that students will face on an 
upcoming state exam — a key hurdle to graduation. 


This school is one of eleven high schools in New York City that are experimenting with Drive to Write, a new 
program in writing instruction and teacher support. This coaching and professional development program for 
teachers is designed and administered by New Visions for Public Schools, an educational support network that 
delivers professional development, data infrastructure, leadership training, certification, and other key services 
to New York City public schools serving students in grades 6-12. This report describes the rollout and impact of 
this ambitious program, which took place during the 2017-18 school year as part of a research study.* (See Box 
1 for more on the pilot year for this program.) The U.S. Department of Education’s Office of Innovation and 
Improvement awarded an Investing in Innovation grant to researchers from MDRC, a nonprofit evaluation 
firm, and New Visions to conduct this study. The research team randomly selected 11 of the 23 New Visions 
schools to implement the Drive to Write program (referred to as “program schools” in this report); the other 12 
schools maintained their current writing and Global History instruction (referred to as “comparison schools” in 
this report). Students who attend these New Visions schools are low-income and high-need with relatively low 
scores on standardized English Language Arts tests. The evaluation and intervention focused on ninth-grade 
Global History classes within study schools.? (See Box 2 for more on the random assignment protocol used for 
this study.) 


The theory of change underlying the Drive to Write program is that students can become better writers through 
regular cycles of drafting and revision, as research studies have shown.? Other studies have shown that teachers 
who receive regular coaching for this type of instruction are better able to improve student writing.* The Drive 
to Write program combines these approaches to improve student writing by helping teachers provide students 
with regular, even real-time, feedback on their draft papers. Free, interactive, online tools — specifically GSuite 
for Education (formerly Google Apps for Education) — make it easier for teachers to quickly distribute assign- 
ments digitally, frequently comment on and grade student essays, and tailor their lessons to meet the needs of 


1 Balu, Alterman, Haider, and Quinn (2018). 

2 In schools with just one Global History teacher, that teacher participated in the program and study. In the few 
schools with more than one Global History teacher, the principal may have chosen among teachers based on who was 
able to attend the professional development sessions. One school had two teachers participate, and one school had 
four teachers participate. 


3 Graham and Perin (2007). 
4 Kraft and Blazer (2018). 
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Box 1 


Incorporating Lessons from Program Pilot to Full Implementation 


New Visions for Public Schools piloted the Drive to Write program in 12 schools during the 2016-17 school year. 
At the launch of the pilot year, the program’s goal was to instill good writing and technology habits in first-year 
high school students who could extend these habits to other courses and subsequent grades. For teachers in 
this pilot year, a key element of Drive to Write was adopting new technology skills that could help students and 
teachers better work together, exchange documents, and revise assignments online. 


The original vision for the program was to first train teachers to use Google tools, then train them to use these 
tools to comment on student writing, and finally train them to use Google tools and Global History content to im- 
prove the draft revision process for students. But many teachers struggled to combine technology, writing, and 
feedback in line with the Drive to Write model. Teachers instead needed more focused help teaching students 
higher-order writing skills, such as developing an argument, as many teachers limited their lessons on writing 
structure to a focus on sentences, paragraphs, and essays. This prompted coaches to recognize that teachers 
needed help teaching writing and providing constructive feedback to their students. In the middle of the pilot 
year, New Visions adjusted its focus to address writing and feedback more explicitly. 


During the 2017-18 implementation year, all three New Visions coaches and nine of fifteen teachers returned 
to the program, giving New Visions the opportunity to enact the following improvements to the program: 


¢ A focus on helping teachers teach effective, higher-order writing skills 
¢ Awriting-specific syllabus for the school year 
¢ Askills-based rubric to encourage the development of specific writing skills and actionable feedback 


¢ Dynamic spreadsheets that use data to track student performance throughout the year 


individual students. Coaches for New Visions teachers provide one-on-one training for online tools and conduct 
group professional development sessions. 


This report begins with an overview of the Drive to Write program and the tools that it offers teachers and students. 
A discussion follows about program implementation and impact in relation to these key findings that pertain to each 
of the program’s main actors--coaches, teachers, and students: 


¢ Coaches tailored their feedback for individual teachers, helping them master technology tools to improve the flow 
of work in their classrooms and cultivate data on student performance to better individualize instruction. The 
program was implemented as planned during the 2017-18 school year. Overall, teachers expressed a high-level of 
satisfaction with the tools and support provided by the Drive to Write program. 


¢ Teachers customized their use of technology tools and writing instruction to suit the needs of their students and 
the constraints of their classroom. Nevertheless, both program and comparison schools had similar practices 
around writing instruction and technology use. Teachers in program schools, however, exhibited a greater under- 
standing of and proficiency with higher-level writing instruction. 
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Box 2 


How the Study Design Relates to Interpreting the Results 


To determine if the Drive to Write program led to improved student writing, MDRC used a randomized control 
trial (RCT) design, considered the gold standard for drawing causal conclusions. In the spring of 2016, schools 
were randomly assigned to receive Drive to Write services, or to continue with their existing school programs, 
which may or may not have included professional development assistance for writing instruction and technol- 
ogy use in the classroom. Twelve schools were assigned to the program group and twelve to the comparison 
group. 


After the end of the pilot year, one of the program schools dropped out of the study because of internal pro- 
gramming changes. During the 2017-2018 implementation year, principals at the remaining 23 program and 
comparison schools determined which teachers, and which of their class periods, could participate in the study. 
As a result, conclusions from the study cannot be applied to all 9th grade classrooms, or all students in the 
study schools because every Global History class and teacher did not participate. 


In addition, some students who started the year in a study classroom transferred to other teachers or schools, 
so the student sample shrank over the course of the year, but at a comparable rate between program and com- 
parison schools. The sample of students for the estimation of program impact includes those students who 
had a score on the end-of-year writing assessment: 807 program school students, 735 comparison school stu- 
dents. Students are nested within classrooms and schools, so the impact is estimated as a weighted average 
of students within schools, which were grouped into blocks based on a prior history of writing achievement. The 
study design thereby accounted for underlying differences between schools in terms of student and teacher 
skills. 


¢ Students throughout the study performed similarly on an assessment test regardless of their exposure to the Drive 
to Write program. It is therefore unclear if the program had a positive effect on student writing after one academic 
year of implementation. Several factors could have dampened early effects, such as parallel growth in writing 
ability and technology use for ninth-graders in both program and comparison classrooms. In addition, the study’s 
sample of schools may have been too small to detect even modest effects. Finally, the results of the assessment may 
reflect student skill at timed test taking (which all schools address), instead of the intervention’s core focus on 
intensive writing composition (on which program schools spent dedicated time). 


The rest of the report describes the program elements in detail, and each finding includes a brief overview of the 
context for the finding and ends with a tip for program design that may be useful to practitioners who are adopting 
or designing a similar intervention. The report concludes with a discussion about challenges in implementing the 
program and opportunities to improve it. 
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Drive to Write: A New Model of Tech- 
Savvy, Data-Driven Writing Instruction 


rive to Write tackles many of the big challenges facing high school teachers today, including knowing when and 
how to offer specific and actionable feedback on writing in core content classes; using technology to streamline both 
teacher and student workflow; and interpreting data on student progress to customize learning for individual students 
and the class as a whole. Drive to Write is unique in its multifaceted approach to these challenges by emphasizing the sus- 
tained professional development of teachers and one-on-one coaching that are hallmarks of the New Visions approach. 


The Challenge of Teaching Students How to Write 


The development of Drive to Write reflects and responds to a larger and longstanding national conversation about 
the importance of writing instruction. In 2003, the National Commission on Writing declared that writing is the 
most neglected of the three R’s — ‘reading, ‘riting, ‘rithmatic.® Although teachers of all subjects are expected to bring 
reading and explanatory writing into their classrooms, particularly following the launch of the Common Core State 
Standards, there are no clear directions for how to achieve this goal.” There is evidence that tailoring writing instruc- 
tion for different subject areas — and coaching teachers appropriately — may be the most effective way of helping 
students read and comprehend texts differently for different subjects. However, this support does not make it easier 
for teachers to both teach content and incorporate instructional routines for reading and writing.® An evaluation of 
the College Ready Writing Program revealed that teachers who receive training on writing instruction are better able 
to improve student writing. The initiative is part of the National Writing Project’s College, Career, and Writers Pro- 
gram.*° However, little is known about the effectiveness of professional development training on writing instruction 
for core subject matter teachers, and this is the target of the Drive to Write program. 


In New York state, recent changes in the state Regents exams, especially those related to history, have emphasized the 
importance of good writing. As of 2016, the New York State Global History Regents exam had a pass rate of only 68 


5 The Every Student Succeeds Act defines “professional development” as “sustained (not stand-alone, 1-day, or short term 
workshops), intensive, collaborative, job-embedded, data-driven, and classroom-focused,” see United States (1966). 

6 National Commission on Writing (2003). 

7 The Common Core State Standards are a set of high-quality standards in English language arts and mathematics that 
outline where students should be at the end of each grade from kindergarten through grade 12. They were developed in 
2009 by the National Governor’s Association for Best Practices and adopted by 41 states, including New York, and the 
District of Columbia. See National Governors Association (2010). 

8 Shanahan and Shanahan (2008). 

9 Corrin et al. (2012). 

10 Gallagher, Arshan, and Woodworth (2017). 
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percent — one of the lower success rates for any Regents exams required for public high school graduation in New 
York State.** In October 2016, the New York State Board of Regents instituted changes to the Global History Regents 
exam, effective June 2019, that would likely make this exam even harder to master. A key change was the addition of 
a new writing task, the Enduring Issues Essay, which requires students to write about a common theme from a set 
of historical documents. The exam is administered to students in the tenth grade after two years of Global History 
instruction. It calls upon students to go beyond a simple recitation of fact to an informed analysis of written materi- 
als with compelling conclusions. The grade on this part of the timed exam makes up almost one-third of a student’s 
overall score. The Drive to Write instructional program was created to help ninth grade teachers address the difficul- 
ties of state tests by helping their students absorb demanding course content, become better writers, and ultimately 
prepare for the challenging essay they will face on the tenth grade Global History exam. 


On the heels of the increased emphasis on writing comes the ever-expanding availability of technology in the class- 
room, especially the free online tools that Drive to Write incorporates. In 2015, 89% of students in eighth grade 
reading classes had access to computers at school; and there had been a significant increase of students who use tech- 
nology “once or twice a week” for in-class reading instruction, as compared to 2013.*? In the past few years, school 
districts have rapidly adopted Google platforms and free G Suite tools in particular — more than half of the nation’s 
primary- and secondary-school students, totaling more than 30 million students, used Google education apps in 
2017.73 But teachers rarely receive targeted support for incorporating such tools into their instructional practice. 
Google instead depends on early adopters to bring their tools into schools. Beyond Google itself, curriculum devel- 
opers are creating interventions that specifically use technology to support writing. Curriculum developer CAST 
created the Writer’s Workbench Project through online apps, and is currently studying it in 10 schools.** As these 
classroom technology options proliferate, districts and teachers face many choices. Technology and professional 
development providers are starting to coach teachers on ways to integrate technology into specific lesson plans to 
promote the consistent and optimal use of these tools and curricula.*® The theory is that one-on-one coaching, like 
that offered by Drive to Write, might be important for initial adoption. 


At the same time, educators face pressure to adopt data-driven decision making in the classroom. Teachers grade 
students on a given assignment and may do so with greater fairness and consistency if they have a rubric or check- 
list. But little is known about how high school teachers use ongoing homework, classroom work, and test scores to 
monitor student trajectories and differentiate instruction accordingly throughout the year.*® Research shows that 
the first step toward being able to do this is facilitating a culture in which open conversations about student data can 


11 New York State Education Department (2016). 

12 National Center for Education Statistics (2015). 

13 Singer (2017). 

14 See CAST, and Vue et al. (2016). 

15 One example of this approach is the Dynamic Learning Project, an effort that partners regionally-based coaches with 
teachers for an eight-week coaching cycle to tackle a teacher-identified classroom challenge, which shows promising 
school-level buy-in after the pilot year, see Bakhshaei, Hardy, Francisco, Noakes, and Fusco (2018). 

16 Brookhart et al. (2016); Marsh, Pane, and Hamilton (2006). 
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feel productive, actionable, and non-threatening.*’ Drive to Write aims to create this culture in the coach-teacher 
relationship, and supports differentiation by monitoring a student’s work throughout the year.*® 


A Toolkit for Teachers 


The Drive to Write program is as much a response to the growing demands on teachers to improve student writing 
in general as it is a targeted attempt to help teachers better prepare students for the challenging writing component 
of the Regents Global History exam. New Visions hired three instructional coaches to support teachers toward this 
end. The Drive to Write coaches worked with teachers in groups and individually to help them improve writing 
instruction by offering regular feedback to students, using technology to track and grade students, and consulting 
data on student performance. Figure 1 describes program elements and roles. To provide teachers with a systematic 
approach by which to integrate these disparate pieces, the Drive to Write coaches drew upon the lessons of the pilot 
year (see Box 2) and created these three distinct tools for teachers to use in the complex ecosystem of a high school. 


¢ A writing skills syllabus. This is a roadmap for writing instruction that complements the general Global History 
syllabus for specific historical periods and concepts. In professional development sessions throughout the year, 
coaches will model the type of instruction and writing skills in the syllabus that can help teachers turn their stu- 
dents into better writers. Coaches emphasize the need for actionable feedback from teachers so that students can 
learn how to revise their work (not just whether or not to revise it).*® The writing syllabus includes four, in-class, 
multiday writing workshops that teach writing skills and familiarize students with the importance of responding 
to teacher feedback. Each workshop culminates in a Drive to Write essay assignment that is similar to the essay 
requirement on the upcoming Regents exam. Teachers like Ms. B. can adjust the pace at which their classes, or 
individual students, can move through the syllabus based on their progress and needs. During one-on-one ses- 
sions, Ms. B.’s coach can help her make these adjustments without jeopardizing a student’s grasp of the core con- 
cepts of the academic syllabus. 


¢ A skills-based rubric. New Visions developed a rubric to identify 19 distinct skills that students should demon- 
strate in an essay (see Box 1). Ms. B had not seen something this detailed before. She could use this not only as a 
grading tool, but also as a teaching and learning tool because it shows a hierarchy of steps that lead to the mastery 
of each skill. For example, on the topic of starting sentence structure, she will lead students from simple sentences, 
to varied sentence structures, to conjunctions for complex paragraphs — which really mattered to her given that 
students were coming into ninth grade with lower than average English Language Arts test scores. The use of 
strong verbs such as identifies, describes, or explains clarify the task at each level of the rubric. In this way, the 


17 Moeller, Seeskin, and Nagaoka (2018). 

18 Turner and Coburn (2012). 

19 Explicit instruction, the practice of teaching specific skills in a logical order, building upon prior skills, and providing 
frequent and immediate feedback to students, — is an effective way to teach adolescents writing. See Archer and Hughes 
(2011); De La Paz and Graham (2002). 
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Figure 1: Drive to Write Stakeholders and Their Activities 


New Visions for Public Schools 
School support network in New York City that: 
¢« Developed the Drive to Write program 


eee eee nee eee = = === 


f . . . . ‘\ 

* — Recruited the schools for this evaluation H Drive to Write Principals \ 

* Hired the Drive to Write coaches 1 Principals from participating =| 

* — Provided the laptops to schools { schools: 

: ° Provide data for the H 

evaluation study that 

includes student testing, ! 

! h t t H 

Drive to Write Coaches alee and siden 

Three experts in technology, writing instruction, * Designate teachers in H 

or who provide ongoing teacher support program schools to 

aie 4 attend monthl 1 

¢ Plan monthly group professional professional : 

development sessions ; ; development sessions at! 

Meet 1:1 with teachers ona biweekly basis \ the school } 
regarding the use of G Suite tools for XN y 


assignments and feedback 
Tailor support based on individual teacher 


Drive to Write Teachers 
Ninth-grade Global History teachers who: 
« Attend monthly professional development sessions 
¢ Meet with a coach 1:1 on a biweekly basis 
* Lead four Drive to Write writing workshops in their classrooms during the 
school year: 
* — Explicitly teach writing skills, paced by the Drive to Write syllabus 
¢ Distribute Drive to Write assignments using Doctopus 
¢ Provide constructive comments to students using Google Docs 
¢ Score assignments with the skills-based rubric using Goobric 
¢ Interpret data on student writing skills to tailor lessons to student 
needs 


Drive to Write Students 
Ninth-grade Global History students who: 
¢ Participate in four Drive to Write workshops during the school year: 

¢ Interact with teachers via comments using Google Docs 

¢ — Draft and revise Drive to Write assignments based on teacher written 
feedback and rubric scores 
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rubric provides Ms. B. and her coach a common vocabulary that relates to writing instruction, and helps her pro- 
vide students with more specific and actionable feedback than she might otherwise give, which will encourage her 
students to make specific revisions.?° The progression of writing skills in the rubric parallels the progression of 
writing skills in the syllabus, so Ms. B will introduce new sections of the rubric as she teaches corresponding skills 
that first focus on basic skills, and turn to more advanced work over time. 


¢ Customized technology tools. New Visions developed two Google add-ons that Ms. B. used in the Drive to Write 
program: Doctopus is an enhanced spreadsheet feature that allows teachers to collect, review, and grade assign- 
ments from Google Docs; Goobric appends rubrics to student work so that students can see their grades in relation 
to ahierarchy of skills. Both of these apps are integrated with G Suite for Education, a free technology platform that 
allows teachers to electronically distribute documents to students in their Google Classroom stream and interact 
with them in comments on Google Docs. These tools are meant to expedite grading for Ms. B., and make the pro- 
cess more transparent for her students. 


Twice a month, Ms. B.’s coach will visit her at school. The visit will include an observation of writing instruction and 
a one-on-one coaching session to discuss her instruction — either before class to refine her lesson, or afterwards to 
reflect upon it. During this coaching session, Ms. B. and her coach will organize and interpret the skills-based rubric 
data on specific student writing skills. To identify appropriate interventions, they will focus on skills that most stu- 
dents have yet to master but are ready to learn with guidance. They will then plan writing interventions focused on 
these specific skills and track a student’s relative growth across assignments. Box 3 explains how researchers tracked 
the implementation and fidelity of this program for Ms. B and Drive to Write peers, as well as comparison teachers. 


Ms. B. was fairly similar to her Drive to Write peers. She was a second-year teacher, though this was her first year 
teaching in New York City public schools. The other Drive to Write teachers had varying experience — three were 
first-year teachers, the others a mix of novice and veteran teachers. Ms. B. was completely new to Google tools at the 
start of the school year, as were the other five teachers who were new to the Drive to Write program. Her experience 
represents the average Drive to Write experience. 


20 Woodard, Magnifico, and McCarthey (2013). 
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Box 3 


Sources of Implementation Data 


MDRC collected and analyzed a variety of data in order to understand the implementation of the Drive to Write 
program. This data breaks down into the following two categories: 


¢ Ongoing program data: New Visions’ data infrastructure collected Google usage data for each program stu- 
dent on four anchor writing assignments throughout the year. 


a 


Assignment distribution: From the Doctopus data, MDRC analyzed when assignments were distributed 
to students and whether the assignments were differentiated. 


Teacher comments: From Google Docs data, MDRC analyzed how many comments teachers made, the 
timeliness of their feedback, and whether students made revisions. 


Rubric scores: From Goobric data, MDRC analyzed student scores on the rubric and tracked the progres- 
sion of skills across assignments. 


Coaching logs: MDRC worked with New Visions to design an online log that coaches would fill out after 
each session with a teacher and used this data to explore the frequency and nature of coach-teacher 
interactions. 


Professional development and artifacts: MDRC observed every group professional development session 
and collected artifacts for analysis. 


e¢ Program and comparison data: MDRC collected more traditional implementation data to capture service 
contrast between program and comparison schools. 


a 


Classroom observations and teacher interviews: Researchers visited four program and four comparison 
schools (one from each random assignment block for each treatment condition) to observe one to two 
Global History classes. Researchers recorded practices and collected artifacts, including photos of the 
classroom, copies of lesson plans, and examples of student work with teacher feedback. Interviews 
captured writing instruction practices, technology use, and support received, as well as perceptions of 
student achievement in writing and teacher perceptions of their own self efficacy in teaching writing. 


Student and teacher surveys: All students and teachers in both program and comparison schools were 
invited to participate in a short survey in the fall and spring. Student surveys captured their experiences 
with writing and teacher feedback in Global History classes, along with student feelings of self efficacy 
around writing. Teacher surveys captured their experiences with teaching writing, using technology to 
facilitate workflow, and their feelings of self efficacy around writing and teaching writing. 
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Inside the Intervention Classroom: 
Findings from the One-Year Evaluation 


MDEC researchers observed 15 teachers in the 11 program schools, and 17 teachers in the 12 comparison schools. 
The analytic sample included 1,008 program students and 936 comparison students. The study took place during the 
2017-2018 academic year, beginning in September and ending in May. This report presents three broad categories of 
findings as they relate to coaches, teachers, and students. 


Coaches Played A Pivotal Role in the Drive to Write Program 


Helping Teachers Use Writing and Technology Tools 


The professional development sessions run by coaches followed the writing skills syllabus as intended, and targeted 
different writing skills each month. Teachers said the sessions helped them understand the skills that students need 
to become better writers. Coaches began each session by reviewing progress on the syllabus, showing how writing 
and technology tools can together help teachers provide students with timely and effective feedback. The use of 
technology was limited to Google tools like Doctopus and Goobric, which facilitated the writing workshop.?* This 
reflects the very essence of the intervention — that good writing can be facilitated by technology, but that technology 
need not be the driving force behind a student’s academic engagement. 


Throughout the year, coaches also responded to teacher needs regarding data use and interpretation, helping to 
manage their work and cognitive loads. Coaches initially asked teachers to use fall writing assessment scores to sort 
students into high, medium, and low skill groups, so that teachers could differentiate, or tailor, their instruction to 
address specific skill gaps for individual students. After the first Drive to Write assignment, when teachers reported 
being overwhelmed by this type of individual-level data analysis and translation into instructional differentiation, 
coaches instead focused on using data to help teachers determine what skills were lacking for the greatest number of 
students in their classes. 


Supporting Teachers’ Individual Needs 


Overall, teachers continued to participate in professional development each month through March, when the last 
Drive to Write assignment was scheduled (14 of 15 study teachers attended in September compared with 12 of 15 in 


21 During the first half of the year, coaches used professional development sessions to demonstrate additional but not core 
technology tools such as: Quill (an online writing skills diagnostic), Spiral (a platform for collecting rapid digital student 
responses in class), and Screencastify (a tool to give video feedback to students). 
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March). The amount of coaching also remained consistent throughout the year. In September and October for the 
first Drive to Write Assignment, coaches made 21 visits across all 15 teachers; for the last Drive to Write Assignment 
in March and April, coaches made 27 visits across all 15 teachers. An analysis of the logs that coaches filled out after 
each formal coaching interaction shows more infrequent coaching input than expected (approximately monthly on 
average, rather than biweekly visits). However, the coaching logs do not capture the informal interactions between 
coaches and teachers that may have been more frequent. Coaches described these interactions in interviews: late 
night emails to assuage the nerves of a teacher before a writing workshop day; midday texts to walk a teacher through 
distributing an assignment on Google Classroom; personalized lesson planning support during lunch breaks; and 
one-on-one time with individual teachers during professional development group sessions. In Ms. B.’s case, she and 
her coach were constantly emailing each other between professional development sessions. These additional interac- 
tions helped to strengthen their relationship and build trust between them. 


Coaches tailored their support to accommodate a teacher’s level of technological fluency. At a minimum, the Drive 
to Write program helped teachers manage their classrooms so that there were clear rules and boundaries for student 
laptop use for academic purposes only.” If a teacher was able to manage the classroom and easily use core technol- 
ogy, the goal was to focus session time on reinforcing writing instruction from professional development sessions, 
and on using data for decision making. 7° In cases where teachers needed support to manage the classroom, coaches 
provided that support instead of jumping to higher-order writing skills training, or data interpretation. 


The Drive to Write program faces an inherent tension between individualizing teacher support services and ensuring 
that all teachers experience the program’s full intervention. To determine the appropriate pace for guiding teachers, 
coaches mirrored the thought process that teachers use to work with their students: When do you need to repeat cer- 
tain concepts to ensure mastery, and when do you push ahead with your pacing? Coaching sessions offered individ- 
ual enhancements and tailored supports. Yet several teachers struggled with aspects of the intervention. In March, 
one-third of teachers still needed support on the mechanics of the intervention — classroom management and dis- 
tributing assignments online. On the other hand, another third of teachers were so advanced that coaches were able 
to help them implement skills that went beyond the scope of the basic syllabus (for example, peer editing, using a full 
rubric to grade students, distributing more assignments). 


TIP FOR PROGRAM DESIGN: In a multifaceted intervention, a one-on-one support structure coupled with a 
gradual introduction of program elements can help teachers implement every intended aspect of a program. 


22 Bingham, Pane, Steiner, and Hamilton (2018). 

23 This approach falls within the framework of technological pedagogical content knowledge, in which teachers combine 
knowledge of technology, pedagogy, and content in order to effectively integrate technology, see Koehler and Mishra 
(2009). 
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Teachers Were Receptive to the Drive to Write Program 


Teachers Customized Their Use of Drive to Write Practices and Technology Tools 


Throughout the school year, teachers maintained the expected pace of their writing syllabus. On average, 91% of 
teachers said the pace of implementing Drive to Write activities in their classrooms was “just right.” Per the syllabus 
and rubric, all teachers started the year focusing on sentence-level interventions during their first writing workshop. 
The coaching logs show that by March, 11 of 15 teachers were working on higher-order skills with students, such as 
writing thesis statements and analyzing documents. During classroom observations in April 2018, teachers were 
reviewing foundational skills while layering in new ones. For example, one teacher asked students to look at doc- 
uments and write a corresponding Enduring Issues thesis statement, but he also challenged students to structure 
their sentences in a way that could hold many ideas at once, recalling earlier sentence-level instruction. Teachers said 
the skills-based rubric was helpful, in part because it provided vocabulary for specific and actionable feedback they 
could provide to students. One teacher said in an interview, “it helps the kids know, “This is what I’m grading you on, 
this is what success will look like,’ and it is nice to be able to track what they’re doing.” Another teacher reflected after 
a professional development session that, “Using a rubric consistently throughout the year helped students become 
accustomed to a series of expectations and standards that drove academic growth.” 


e As Familiarity with the Program Grew, Teachers Often Increased 
Their Use of Program Interventions 


Program teachers found the structure and frequency of writing workshops to be valuable because it allowed for deep 
engagement in writing processes. On surveys, program and comparison students indicated that they used similar 
writing processes in class, including brainstorming, outlining, and revising work; and levels for both groups of stu- 
dents were similar at the beginning and end of the year. But interviews reveal that program teachers went beyond 
basic writing processes. In comparison classrooms, teachers addressed writing skills in one-off lessons, or only once 
during the year, but in program schools, the writing workshop model gave repeated, dedicated time for skills instruc- 
tion, in a sequence that allowed students to build their writing skills from one workshop to the next. At the start of 
the year, teachers were hesitant to dedicate the suggested five days of classroom time for the writing workshop, con- 
cerned about minimizing time for content instruction in favor of writing activities. But as the year progressed, some 
teachers increased the duration of the writing workshop, extending time for instruction and composition by several 
days. By the March professional development session, coaches suggested thinking about the writing workshop in 
phases, rather than days. While instruction would still cover the same elements of engaging with evidence, skills 
practice, writing, and assessing student work, this new approach encouraged teachers to adapt the workshop timing 
to the needs of their own classroom. Ms. B. used seven days for her last Drive to Write workshop. 


TIP FOR PRACTICE: Providing teachers with guidance, and anchoring writing instruction in specific skills that 
can grow progressively stronger, with dedicated class time for developing proficiencies, can give both teachers and 
students clear expectations for progress. 
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e Teacher Comfort with Online Tools Varied 


Teachers varied in their use of Drive to Write tools for electronic assignment, distribution, and grading. All program 
teachers began the year regularly using program tools, but this usage waned during the year for some teachers. (See 
Figure 2.) For example, on assignment 1, all 15 teachers distributed the assignment using Doctopus. However, in the 
second half of the year, four of these teachers (teachers L through O in Figure 2) did not distribute assignment 3 or 
assignment 4 at all. This could suggest pacing issues (running out of time at the end of the year), competing prior- 
ities, or teacher turnover. In addition, Figure 2 also shows the proportion of students who received feedback from 
their teachers (see the bars in Figure 2) as well as the average number of comments per student (see the numbers 
beside each bar in Figure 2). Teachers A through G provided feedback to all or nearly all of their students for each 
assignment, while teachers I through K provided only some of their students with feedback on each assignment. This 
variation suggests that the latter group of teachers may have been trying to focus their time and energy on specific 
students. This demonstrates a depth, not a breadth, of practice that evinces a triage grading strategy. 


Even though all teachers did not distribute every assignment, those who did and those who also provided feedback 
generally offered more comments per student as the school year progressed. (See the numbers to the right of each 
bar in Figure 2 for the average number of comments per student on an assignment.) Overall, across all teachers, the 
average number of comments per student increased from 4.2 on assignment 1, to 6.1 on assignment 4. 


Similarly, teacher use of the online rubrics changed over the course of the four Drive to Write Assignments (see 
Figure 3).?* For example, for the first Drive to Write Assignment, more than half of the teachers were completing the 
rubric for at least some of their students across all writing skills categories (dark or light blue segments of the rubric 
bars). However, by the time Drive to Write Assignment 3 was administered during the second half of the school year, 
not only did the use of the rubric decrease, but there was also a shift in grading. At least 25 percent of teachers were 
not grading any of their students using Goobric for assignments 3 and 4 (as seen by the grey portion of the bars in 
Figure 3). In addition, the proportion of teachers grading the other four writing skills for all of their students gener- 
ally decreased from assignment 1 to 4 (the shortening of dark blue segments over time). 


In a positive shift over time, more teachers began providing feedback on the higher order skill of analysis (see Figure 
3). These data also suggest that teachers may have differentiated grading by writing skills, but not all teachers were 
able to maintain that throughout the course of the school year. 


TIP FOR PROGRAM DESIGN: Support from a coach should be both nimble, and faithful to core technology prac- 
tices. This approach can better accommodate teacher adaptations of online tools due to time constraints and student 
needs. 


24 The Goobric tool used the five main writing skills domains from the skills-based rubric — Sentence Structure, Para- 
graph Structure, Evidence Integration, Essay Structure, and Analysis. The first three writing skills have three measures 
within them, Essay Structure has four measures, and Analysis has six measures. Figure 3 shows that teachers graded more 
than half of their students on sentence structure (see the dark blue bar) in assignments 1 to 3. The teal portion of the bar 
indicates that teachers graded one or two measures for around ten percent of students, and around a quarter of the stu- 
dents received no grade on any of the sentence structure measures. 
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Figure 2: Percentage of Students who Received Comments or Feedback and 
Mean Number of Comments on Each Drive To Write Assignment, by Teacher 
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SOURCE: Data collected by New Visions for Public Schools from Google Classroom. 


NOTES: The percentages shown in this figure are among those teachers who administered each Drive to 
Write assignment via Google Classroom. 

The number beside each bar represents average of the number of teacher comments and replies to those 
comments by peers and teachers, as well as teacher edits that each student received. The number of 
comments represent the average across all of a teacher's course sections. 


SUPPORTING TEACHERS WITH THE DRIVE TO WRITE PROGRAM 


Figure 3: Teachers Using Rubrics for Their Students, by Skill, for Each Assignment 
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SOURCE: Data collected by New Visions from Google Classroom. 


NOTES: This figure shows Goobric data for a total of 15 teachers along with the proportion of students for whom they 
used this online tool. The Goobric tool uses five key domains of writing skills from the skills-based rubric — sentence 
structure, paragraph structure, evidence integration, essay structure, and analysis. Each writing skill has different 
measures that a teacher can grade for their students. The first three writing skills have three measures, essay 
structure has four measures, and analysis has six measures. 
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Teachers in Program Schools Developed a Deeper and More Nuanced Understanding 
of Writing Instruction and Technology Tools 


Drive to Write answers a broader call for improved writing instruction and strategic technology use in the class- 
room. In turn, both program and comparison teachers were eager to add both elements to their classrooms during 
this study. New Visions and the local school district offered peer-led professional development training for all Global 
History teachers, including summer and school year sessions, on both history content and the G Suite tools. Some 
schools already had instructional coaches available for teachers across subjects and skills. Google’s free G Suite tools 
provided an accessible and convenient technology choice for both program and comparison teachers, who may have 
been seeking these tools on their own.?® But the depth of understanding about how to best incorporate technology 
and writing instruction in the classroom differed between study groups. 


¢ G Suite Technology Tools Became Part of Standard Instruction Across Schools 


The importance of teachers feeling comfortable with new technology is central to the success of the Drive to Write 
workshops because they operate on a digital platform. In general, comfort with technology is a critical hurdle for 
teachers to clear. Dismissive attitudes and beliefs about technology and limited knowledge and skill can impair the 
efforts of many teachers to adopt new technology. In contrast, a positive embrace of technology may make teachers 
more likely to use it.2® Those teachers who found the G Suite apps difficult at the beginning of the year, continued to 
struggle. This was due to a lack of basic technology literacy and inexperience, rather than a lack of interest. Of one 
teacher, a coach said: “I don’t think it’s that [the teacher] didn’t want to know. I think that there’s just steps before- 
hand that need to take place.” The coach went on to suggest that a course in navigating Google Drive or how to use 
Google Docs might be “prerequisite to do some of the things that we’re asking teachers to do in Drive to Write.” 
For those who were proficient in technology, the G Suite tools facilitated the flow of assignments and feedback. One 
teacher reflected on G Suite tools at the end of the year, “It keeps me organized — which means that I don’t lose [stu- 
dents’] papers, but rather they actually get the feedback. Students feel like they are being more supported and held 
accountable when they see feedback on their work, and I see more willingness to edit/revise now.” 


Given that many technology tools are widely advertised and free, many teachers are likely to be aware of and some- 
what comfortable with technology even without formal training. As a result, technology and G Suite tools often per- 
meated both program and comparison classrooms and lesson plans. A survey conducted in spring 2018 for this study 
found that, on average, teachers in comparison schools reported using computers or laptops in their classrooms for 
monthly instruction, compared to weekly instruction for Drive to Write teachers. Comparison teachers received lit- 
tle coaching on technology for instruction but sought support from tech-savvy colleagues such as English Language 
Arts teachers, who were more experienced in using technology to facilitate writing instruction in the classroom. One 
comparison school teacher who recently began using G Suite tools for writing said: “I’m finding that I wish I had 
spoken to the English teacher sooner. I wish I wasn’t so afraid of Google Docs previously. I’m not very comfortable 
with it, but it was very easy. And I think the kids were getting more out of it and now maybe my frustration of them 


25 Yim, Warschauer, and Zheng (2016). 
26 Ertmer et. al. (2012); Hutchison and Reinking (2011). 
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not reading my feedback will disappear.” All of the comparison teachers who were interviewed for this study said 
their schools did not advance efforts by administrators to secure more technology in the classroom. 


TIP FOR PROGRAM DESIGN: Training teachers in the fundamentals of technology is an important pre-requisite 
for a technology-based intervention. 


e Program Classrooms Emphasized Higher-Level Writing Skills than Comparison Classrooms 


Both program and comparison schools focused lessons on the new Regents Enduring Issues Essay. In both sets of 
classrooms, teachers referred to the essay writing task during instruction and hung posters on the walls defining and 
discussing various enduring issues. However, program teachers covered higher-level skills such as analysis, while 
comparison classes focused on more mechanical skills of sentence and paragraph construction by the spring of 2018, 
according to research team observations in person and via student assignments at the time. 


Figure 4 illustrates how program students demonstrated these advanced writing skills, and how teachers provided 
specific skills-focused feedback. In the third Drive to Write assignment in March, the teacher in Figure 4 is address- 
ing skills related to analysis, a higher-order skill. The teacher reminds the student of sentence skills taught earlier 
in the year, reinforcing success and providing specific suggestions about analysis skills that still need improvement. 
The feedback reflects language from the rubric, anchoring both teacher and student in the specific skills. Program 
teachers who were observed in April also used the rubric language during class instruction, as in this assignment: 
“The thesis statement should outline every paragraph. You may need 3 examples with a comma to describe every 
body paragraph.” The technology-assisted workflow also contributed to more robust feedback. As one teacher said, 
when they spend less time flipping through papers and hand-writing comments, “it’s the quality of the feedback” 
that improves. 


Meanwhile, also in March, comparison teachers were still focused on basic skills: vocabulary, syntax, grammar, 
and writing complete sentences. In one comparison school, a lesson that was supposed to be about students revising 
their own work, instead focused on line edits, and spelling corrections rather than how to craft stronger writing and 
arguments. Another comparison teacher gave a student verbal feedback and did not use academic language or rubric 
skills. Instead of asking for “evidence,” the teacher simply applauded the student’s use of examples without utilizing 
academic language and providing feedback for improvement. 


This difference in skill focus is also evident in the inability of comparison teachers to help students compose a full 
essay. One comparison teacher said, “I’m not a hundred percent sure how to do this, to get them to take what they 
know about social studies and connect it to how to write it down.” A visit to one comparison school showed students 
submitting handwritten compositions. One student’s work demonstrated a cohesive body paragraph but did not 
integrate any outside documents or analysis of evidence. Yet the teacher provided just one piece of feedback scribbled 
in the margin: “Needs additional info with analysis,” which may be vague to a student. Even though there may have 
been more useful verbal feedback, the student was without any written record of the specific exchange for reference 
during later revision or grading. 
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Figure 4: Example of Student Work with Teacher Comments for Assignment 3, 
with Teacher Feedback Strategies and Writing Skill Types 


Body paragraph 1 (first thesis part): 
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document one it states “One 
positive, undisputed result of the Crusades was a greatly expanded knowledge of geography 
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between East and West, it is plain now that each side made major contributions to the culture 
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Body paragraph 2 (second thesis part): Trade between the Americas, Europe, and Africa was —— 

known as triangular trade. Christopher Columbus sailed on the Atlantic Ocean and landed in 

the West Indies in 1492. According to document three it states “His voyage led to the ——<—— « 
colonization of the Americas by European countries like Portugal, England, Spain, and Mar 2, 2018 

France. Trade between the Americas, Europe, and Africa became known as triangular trade." 

Christopher Columbus's voyage was the cause of the colonization of the Americas by pathy aching agama 
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SOURCE: New Visions for Public Schools. 


TIP FOR PROGRAM DESIGN: To help students compose a full essay, lesson plans should require teachers to name 
specific writing skills as part of their feedback for students, and teachers should be required to introduce skills in an 
explicit hierarchy and sequence. 


e Data Helped Program Teachers Advance Instruction 


Using detailed student data on writing ability to inform instruction was new for the Drive to Write teachers and 
difficult for teachers to do on their own. Although Google Classroom and Doctopus could theoretically make the 
distribution of differentiated assignments easier, by the time of the second Drive to Write Assignment in December, 
teachers shied away from creating different assignments for different students. In April, one teacher said it would 
be “unrealistic” to use data for differentiation every day because it would take too much time to administer assess- 
ments and interpret results. This same teacher, however, said that student writing skills data from the Drive to Write 
assignments helped her focus classroom instruction for certain students. A coach echoed this trend across teachers: 
“What we found was that as we got to Drive to Write [assignments], especially [pumbers] two, three, probably four, 
teachers were differentiating their instruction, not necessarily their materials.” Teachers used scores from the writing 
skills rubric to guide lesson planning and skills development for entire classes of students and talked with their coach 
about interpreting Goobric data. 


Prior to the program, teachers might have used the Regents rubric to grade students in writing, but that scoring 
system does not rate specific skills or include a tool to compile scores across students into a comprehensive dataset, 
making it difficult to look for trends in achievement across students. With Drive to Write tools and coaches, though, 
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teachers could test their own instruction — pick out specific skills to target, run an intervention on the skill for one 
writing workshop, and check the result after grading the next assignment. One teacher reflected that “there are still 
students who are still struggling. And that’s something that I would need to continue addressing in [the next Drive to 
Write assignment]. And I’m planning, ... based on that, to make that very fast review with them and continue giving 
them feedback.” When coaches guided teachers in using data by preparing spreadsheets, for instance, teachers were 
able to interpret and use student data. Although most program teachers were convinced of the value of skills data at 
the student-level and assignment-level, coaches found that very few teachers reviewed or used data from Goobric on 
their own. 


Comparison school teachers said they based their writing instruction more on general perception than specific skills 
or data. One comparison teacher said: “Whenever I get the [writing rubric] data back from my students I know what 
they need to work on, so it does help. But I would say... I think it’s more of a general sense when I’m grading their 
work and then I see how they’re doing. That’s how I usually drive my lessons.... I always grade their work and I put it 
in, but I don’t think about it.” Although this teacher was able to identify the need for data, she lacked the opportunity 
to incorporate data in her instruction. 


TIP FOR PROGRAM DESIGN: Teachers who are interested in using student data need ongoing, intensive support 
to help them access, interpret, and act on it. 


e Drive to Write Turned Changes in the Regents Exam into an Opportunity for Teachers to Focus 
on Writing Education 


An important change in the Global History Regents exam was that it only tests content that is covered in tenth 
grade.?’ Asa result, ninth grade teachers were under less pressure to focus on history content and could redirect time 
to improving writing skills. This was particularly evident to Drive to Write teachers. One program teacher said: “... if 
I have more autonomy to spend more time on these [writing skills], which it seems like my school supports, then ’'m 
going to use the more time. Because it’s not how much content that I get through anymore, it’s how can I teach this 
enduring issue.” On the other hand, a comparison teacher said, “And I’m always torn between, okay I have to teach 
them how to write an essay, but I also have to teach them about Rome. So it’s like how. How do I do that in an effective 
way under a time crunch? It’s tough.” This suggests that Drive to Write helped reframe these curriculum shifts into 
opportunities for teachers, rather than additional burdens. 


TIP FOR PROGRAM DESIGN: Teachers may be better able to integrate new practices and teach new skills during 
academic years without high stakes testing. 


27 The Global History curriculum is taught across ninth and tenth grade, with the exam at the end of the tenth-grade 
year. However, starting for those in ninth grade in the 2017-18 school year, the Global History Regents exam would cover 
content only from the tenth-grade curriculum. 
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It Is Unclear if Student Writing Benefited from the 
Drive to Write Intervention 


Assessment Scores for Program and Comparison Schools Showed 
No Significant Difference 


In the initial design and pilot phase, the Drive to Write program expected to cross subjects, from Global History to 
English, thereby exposing students to two years of writing instruction via technology. The program’s initial focus on 
feedback and using technology to facilitate feedback and revision (see Box 1), had an intensive teacher development 
focus. The evaluation proposal for Drive to Write initially assumed a moderate to large effect based on factors such 
as a two-year exposure to the program, multiple occasions for writing practice and feedback (particularly in program 
schools), and the results of previous research that were focused on the role of feedback in experimental or quasi-ex- 
perimental studies.?® The expected magnitude was equivalent to one to two months of learning (if writing effect 
sizes are comparable to reading).?° The study team recruited a number of schools that would have been required to 
detect the expected effect. The outcome measure was policy relevant: a timed essay writing task that is similar to the 
Regents essay.?° 


At the one-year point of actual implementation in one subject (instead of two years in two subjects), there are fewer 
differences between program and comparison schools than expected. Comparison schools, as noted earlier, also 
seem to be using technology and focusing on writing. Comparison school students also reported the same high 
levels of self-efficacy related to writing as their program school counterparts on fall surveys (3.7 on a scale from 1 to 
4), leaving very little room for growth over the year. Indeed, by the spring surveys, both program and comparison 
students reported very little, if any, growth relative to the beginning of the year in terms of the perception of their 
writing ability. 


28 The feedback-focused research from the 1990s and 2000s did not study interventions that integrated the use of instruc- 
tional technology. 
29 Initial expected minimum detectable effect size was between 0.27 and 0.45, varying based on the number of students 
per school but assuming a two-tailed test at power of 0.8, intra-class correlation of 0.2, and an R-squared of 0.5 at the stu- 
dent and school levels. The effect size translation into months of learning is based on estimates for reading tests; see Hill, 
Bloom, Black, and Lipsey (2007). 
30 The writing assessment was graded using the Regents Exam rubric for the prior Thematic Essay section. Scoring was 
broken up into the following categories and students could be given a score from 1-5, reflecting the extent to which students: 

1. address the task (task subsection) 

2. utilize higher thinking skills in their response (analysis subsection) 

3. use the information provided in the documents (documents subsection) 

4. support their ideas with facts, examples and details (facts subsection) 

5. organize and develop an essay (organization subsection) 

6. incorporate relevant outside information in their essay (documents subsection) 
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By the spring assessment, the program had not demonstrated an impact. In terms of raw growth in student scores 
on the timed writing task between the fall and spring, students in both program and comparison schools improved 
their scores across all six subsections of the assessment (see Figure 5). When the research team controlled for student 
and school characteristics, students’ fall scores, and students nested within schools, the analysis finds little detect- 
able difference in spring writing scores between the program and comparison schools. The overall score (out of 30) 
for students in the program group was 15.9, and 15.3 for the comparison group. This difference is not statistically or 
practically significant. Figure 6 shows that the estimated differences are not precise — the 95% confidence intervals 
are large and include an estimated difference of zero, which is likely related to having too few schools in the sample. 
Therefore, it is inconclusive whether this intervention boosts student scores or not. If the true effect was as small as 
the average effect shown in each of the squares in the middle of each line of Figure 6, a larger sample of twice the 
number of schools that were in this study would be required to detect such a difference. 


It is difficult to find a clear point of comparison between this intervention and others focused on writing, to know 
whether one should expect a larger effect size than what was observed. Other evaluations of writing interventions 


Figure 5: Raw Scores on Pre- and Post-Literacy Assessments with Overall 
Test Scores, by Assessment Category 
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Figure 5 (continued) 
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SOURCE: MDRC calculations based on student records obtained from New Visions Public Schools. 


NOTE: Unweighed means reported in this table are based on the analytic sample, which includes all students who 
had post-assessment data in study schools for both fall and spring semesters. 
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Figure 6: Impact Estimates and 95% Confidence Intervals of the Impact Estimates 
on Six Sub-Sections of the Post-Assessment for the Analytic Sample 
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SOURCE: MDRC calculations based on student records obtained from New Visions Public Schools. 


NOTES: The analyses reported in this figure are based on the analytic sample, which includes all students who had post- 
assessment data in study schools for both the fall and spring semesters. 

The red square is the estimated effect of the program on each test section. The line indicates a 95% confidence interval 
for the estimated effect. 

Estimated impacts are regression-adjusted using ordinary least squares, controlling for random assignment blocks by 
school, as well as the following baseline characteristics: race, gender, free or reduced-price lunch status, English as a second 
language, qualification for a gifted program, and a baseline measure of the outcome variable. 

Estimated impacts are based on a two-level model with students nested within schools, controlling for random assignment 
block and school- and student-level covariates. “DtW Schools” are the observed mean outcomes weighted by the number of 
program schools in a random assignment block, and the values for“Non-DtW Schools” are regression-adjusted mean 
outcomes for the non-DtW schools using the mean covariate values for students in DtW schools as the basis for the 
adjustment. 

Student level covariates include: students' race, IEP status, ELL status, gender, 8th grade reading and math test scores, 
fall pre-assessment scores. 

School level covariates include: average score on the thematic essay question on the 2016 and 2017 Global History 
Regents exam (note: 2 schools are missing average regents score in 2016), charter status, percentage of English language 
leaners and students with an IEP, and average attendance rate (the last three measures are from the 2016 school year, just 
prior to random assignment). 

Effect sizes were computed using the standard deviations of all non-DtW school students for the respective measures. 

A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** =5 percent; * = 10 percent 

Effect sizes are calculated by dividing the impact estimate by the standard deviation of the outcome measure for students 
in the analytic sample who are in the non-DtW school group. 

Rounding may cause slight discrepancies in calculating sums and differences. 

None of the results shown the the figure are statistically significant. 
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often focus on just one skill or aspect of writing, while this study’s outcome evaluated a broad set of writing skills 
(as suggested by the skills-based rubric for teachers and students).** None of the field experiments related to writing, 
and reviewed by MDRC, incorporated technology, though there are reasons to expect that technology could either 
amplify or reduce effects. More recent interventions that incorporate technology in instruction are largely focused 
on students with special needs, and for a shorter duration of a few months but more intense usage during that time. *? 
There is research on the effect of professional development and coaching interventions on student literacy achieve- 
ment. Group training for teachers has positive effects (equivalent to almost a month of student learning), but smaller 
coaching programs increased the effectiveness (equivalent to two to three months of student learning). In short, the 
elements that make Drive to Write innovative, also make this study’s results hard to compare to other evaluation 
findings. 


Summary of Findings and Implementation Factors Related 
to Program Impact 


Overall, Drive to Write rolled out the multiple layers of its intervention as intended. Drive to Write coaches worked 
with individual teachers to help them use program tools at their own pace, ensuring a high level of teacher partici- 
pation in professional development and coaching sessions with full adherence to the program protocols laid out in 
the syllabus. Although teachers in program classrooms began the implementation year by regularly using the G Suite 
feedback function, it was often used inconsistently or had a different focus by year’s end. However, tools were gener- 
ally adopted, and teachers appreciated both the tools and the support. At first, the differences between program and 
comparison classroom practices appeared slight but Drive to Write teachers did develop and report a more robust 
understanding of the important roles that feedback, technology, and data can play when teaching students how to 
write. 


The scope of this initial Drive to Write implementation was limited in terms of both duration and sample size, which 
may explain why there were no obvious differences in outcomes for program versus comparison school students. 
Regardless of intended duration or intensity, some of the challenges faced in creating new instructional routines 
reflect the circumstances of working in ever-changing high schools, particularly schools serving high-need students. 
Drive to Write faced teacher turnover (one school losing its only study teacher in October, one school losing its only 
study teacher in May, and another school losing its only study teacher in January and again at the end of the year); 
changing student rosters; and spotty attendance. These circumstances pose a challenge for teachers and students in 
terms of creating both technology and writing routines. In addition, these circumstances made it hard for teachers 
to track their students’ progress over time. Furthermore, schools that are on a trimester system or that offer bilingual 
education, may not be able to follow the same pace of program implementation as teachers in other classrooms. As 
a result, some teachers may need to rely more heavily on extra coaching to tailor professional development services 
to their specific needs. 


31 Graham and Perin (2007). 
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It is possible that program teachers need more time to master Drive to Write’s many intervention components, and 
when they do, the difference between program and comparison student scores may be larger. It is also possible that 
effects could differ if there were a greater number of teachers within a school offering this type of writing instruction, 
to increase alignment between subjects and grade levels, and to support each other in implementation. In addition, 
more months of student exposure could support students in their developmental trajectory as critical thinkers and 
writers. Future programs and evaluations may want to test additional time, focus on one component that has deliv- 
ered large writing effects in the past, or include a larger sample size of schools to detect smaller effects. 


These questions come at a time when the field of education is grappling with program design questions for a variety 
of interventions, including: How much growth in student performance is natural to expect in ninth grade and how 
much more intensive does an intervention need to be to demonstrate impact? How long should students be exposed 
to an intervention before we can expect their skills to start changing noticeably? How refined or masterful does 
teacher instruction and skill need to be in order to produce student-level effects? Does it take longer for programs 
with many components to show effects, or can they deliver change sooner than one-dimensional interventions? 
These are questions New Visions is exploring as well. 
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The Future of Effective 
Writing Instruction 


EF” Ms. B., her school has agreed to implement the Drive to Write program for a second year so she will continue deep- 
ening her skills in writing instruction. The students in her current, 2017-2018 class will move on to tenth grade with a 
new teacher for their final year of Global History. They will then focus more intently on learning content in preparation 
for a spring Global History Regents Exam that will test their knowledge and understanding of historical fact. The exam 
will also include the Enduring Issues Essay that they prepared for in Ms. B’s ninth grade writing workshops. 


The expense of Drive to Write’s intensive coaching and professional development approach may make the program 
hard to scale as is.*? However, there are opportunities for pieces of the intervention to be implemented in the schools 
that have already participated. One school adopted the skills-based rubric in both writing and history classes and 
plans to use it as the basis of school-wide conversations about data. Other schools are building their capacity through 
a train-the-trainer model that involves teachers who either participated in the Drive to Write program, or who are 
tenth grade Global History teachers, among others. New Visions is also exploring ways to harness the power of 
webinars for professional development and online communities of practice as a continuation of their professional 
development sessions for teachers within their network and beyond. In addition, New Visions is redesigning a data 
tool, Goobric+, which had a soft-launch in January 2018. The tool allows teachers to easily compare data across 
assignments and distribute versions of them that are differentiated for individual students. 


The Drive to Write intervention ultimately raises questions about evaluating programs that create new and more 
frequent types of interaction between teachers and students. For example, future research might explore the content 
of the comments that teachers provide to students. This study was only able to advance a high-level understanding of 
the quality of teacher feedback based on teachers’ use of the skills-based rubric and the observations of researchers. 
A deeper analysis or mapping of the content of teacher comments may yield greater insight into what constitutes 
effective feedback and how it may affect student writing. Future research should also consider the length of imple- 
mentation, as a two-year impact finding might differ from a one-year finding. While Drive to Write took place in 
Global History classrooms, other content area classrooms may benefit from similar, frequent, high-quality feedback 
cycles between teachers and students. Students and teachers may also benefit from a more uniform application of 
these practices across subjects. This approach can help students establish more consistent expectations for good aca- 
demic work, and lead teachers to develop a common language for monitoring student progress. 


The one-year implementation of Drive to Write makes clear that a multifaceted intervention requires significant 
effort. Policymakers seeking to introduce a similar program in their school or district should consider the many 
responsibilities of the coaches, the required technology infrastructure, and the demands on teacher time for profes- 
sional development and implementation in the classroom. Understanding these parameters and preparing adequate 
support structures in response to them can lead to the robust implementation of a program like Drive to Write and, 
potentially, significant improvements in student writing. 
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About MDRC 


MDEC is a nonprofit, nonpartisan social and education policy research organization dedicated to learning what 
works to improve the well-being of low-income people. Through its research and the active communication of its 
findings, MDRC seeks to enhance the effectiveness of social and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los Angeles, MDRC is best 
known for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. Its projects are 
a mix of demonstrations (field tests of promising new program approaches) and evaluations of ongoing government 
and community initiatives. MDRC’s staff members bring an unusual combination of research and organizational 
experience to their work, providing expertise on the latest in qualitative and quantitative methods and on program 
design, development, implementation, and management. MDRC seeks to learn not just whether a program is effec- 
tive but also how and why the program’s effects occur. In addition, it tries to place each project’s findings in the 
broader context of related research — in order to build knowledge about what works across the social and education 
policy fields. MDRC’s findings, lessons, and best practices are shared with a broad audience in the policy and practi- 
tioner community as well as with the general public and the media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy areas and target pop- 
ulations. Once known primarily for evaluations of state welfare-to-work programs, today MDRC is also studying 
public school reforms, employment programs for ex-prisoners, and programs to help low-income students succeed 
in college. MDRC’s projects are organized into five areas: 

¢ Promoting Family Well-Being and Children’s Development 

¢ Improving Public Education 

e Raising Academic Achievement and Persistence in College 

¢ Supporting Low-Wage Workers and Communities 

¢ Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United Kingdom, MDRC con- 


ducts its projects in partnership with national, state, and local governments, public school systems, community 
organizations, and numerous private philanthropies. 


